Skip to content

feat(rdma): add worker facade and registered memory ownership - #39

Merged
thweetkomputer merged 1 commit into
eloqdata:mainfrom
thweetkomputer:feat/rdma-memory-registration
Nov 11, 2026
Merged

thweetkomputer merged 1 commit into
eloqdata:mainfrom
thweetkomputer:feat/rdma-memory-registration

Conversation

@thweetkomputer

Copy link
Copy Markdown
Contributor

Add the phase-two RDMA facade and registration layer. Applications can explicitly acquire a worker/context-scoped protection domain, register caller-owned memory, and retain checked slices and operation leases without activating DPDK/SPDK.

Implementation

  • Add BYCORF_ENABLE_RDMA, defaulting to off; only enabled builds discover and link libibverbs and librdmacm.
  • Privately compose RdmaBackend<Worker> with the existing shared ring for descriptor readiness and cancellation. Construction does not access devices, and no additional ring, thread pool, or process-wide blocking mutex is introduced.
  • Lazily acquire PDs for the actual verbs context, distinguish workers and contexts on the same device, and unwind partially acquired resources on initialization failure.
  • Add RegisteredMemory, registration-retaining slices, SGE leases, explicit access permissions, and checked RemoteMemoryRegion{address, length, rkey} descriptors. Validate QP context/PD, permissions, ranges, and SGE length before producing a lease.
  • Document that the final registration reference deregisters the MR before releasing PD/context; callers retain backing memory and quiesce remote access. Multiple QPs in the same PD can share an MR, independently of IMM operations. Storage RegisterBuffers remains a separate registration contract.

This phase provides registration and readiness primitives; connected transfer/IMM protocols and application buffer pools are outside its scope. Slice bounds and exported permission subsets do not restrict the hardware rkey beyond the original MR grants.

Validation

Both configurations used Clang 18, Debug, and BYCORF_ENABLE_WERROR=ON on Linux ARM64. The RDMA configuration additionally enabled BYCORF_ENABLE_RDMA. DPDK/SPDK stayed at their default disabled setting.

cmake --build /tmp/bycorf-no-rdma-stage2 -j 8
ctest --test-dir /tmp/bycorf-no-rdma-stage2 --output-on-failure
# 110/110 passed

cmake --build /tmp/bycorf-rdma-stage2 -j 8
BYCORF_RDMA_TEST_DEVICE=rxe_bycorf0 ctest --test-dir /tmp/bycorf-rdma-stage2 --output-on-failure
# 119/119 passed

The enabled run includes real Soft-RoCE registration/deregistration, real QPs sharing a PD, permission/range/PD checks, device-free failure injection and lifetime tests, and shared-ring readiness/cancellation. Default executable linkage contains no RDMA libraries. clang-format 23.1.1 and git diff --check passed.

@coderabbitai

coderabbitai Bot commented Nov 11, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1cd5c228-576d-4fe7-83ef-3555926e1ae7
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@thweetkomputer
thweetkomputer merged commit 150a50f into eloqdata:main Nov 11, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant